Human Evaluation of Grammatical Error Correction Systems
نویسندگان
چکیده
The paper presents the results of the first large-scale human evaluation of automatic grammatical error correction (GEC) systems. Twelve participating systems and the unchanged input of the CoNLL-2014 shared task have been reassessed in a WMT-inspired human evaluation procedure. Methods introduced for the Workshop of Machine Translation evaluation campaigns have been adapted to GEC and extended where necessary. The produced rankings are used to evaluate standard metrics for grammatical error correction in terms of correlation with human judgment.
منابع مشابه
Grammatical Error Correction of English as Foreign Language Learners
This study aimed to discover the insight of error correction by implementing two correction systems on three Iranian university students. The three students were invited to write four in-class essays throughout the semester, in which their verb errors and individual-selected errors were corrected using the Code Correction System and the Individual Correction System. At the end of the study, the...
متن کاملThe Impact of Immediate Grammatical Error Correction on Senior English Majors’ Accuracy at Hebron University
This study aimed at investigating the effects of grammatical error correction on EFL learners’ accuracy. Twenty-two male and female senior students were chosen randomly to respond to a questionnaire investigating their beliefs about immediate grammatical error correction. Actually, the study was conducted in order to answer this question: what is the effect of grammatical error feedback on stu...
متن کاملThe Impact of Immediate Grammatical Error Correction on Senior English Majors’ Accuracy at Hebron University
This study aimed at investigating the effects of grammatical error correction on EFL learners’ accuracy. Twenty-two male and female senior students were chosen randomly to respond to a questionnaire investigating their beliefs about immediate grammatical error correction. Actually, the study was conducted in order to answer this question: what is the effect of grammatical error feedback on stu...
متن کاملThere's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction
Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metr...
متن کاملAutomatic Annotation and Evaluation of Error Types for Grammatical Error Correction
Until now, error type performance for Grammatical Error Correction (GEC) systems could only be measured in terms of recall because system output is not annotated. To overcome this problem, we introduce ERRANT, a grammatical ERRor ANnotation Toolkit designed to automatically extract edits from parallel original and corrected sentences and classify them according to a new, dataset-agnostic, ruleb...
متن کامل